Skip to content

Phi4 mini#1949

Merged
Borda merged 19 commits intoLightning-AI:mainfrom
ysjprojects:phi4-mini
Apr 3, 2025
Merged

Phi4 mini#1949
Borda merged 19 commits intoLightning-AI:mainfrom
ysjprojects:phi4-mini

Conversation

@ysjprojects
Copy link
Collaborator

New Phi 4 Mini Instruct 3.8B language model.

See issue #1948

  • Outperforms larger models in text-based tasks, including reasoning, math, coding, instruction-following, and function-calling. (e.g. beats Qwen2.5-7B-Instruct in HumanEval)
  • Supports 128k context size.

@Borda Borda added the enhancement New feature or request label Mar 12, 2025
@Borda
Copy link
Collaborator

Borda commented Mar 20, 2025

@ysjprojects mind have look at the failing tests:

FAILED tests/convert/test_hf_checkpoint.py::test_convert_hf_checkpoint[phi-2] - KeyError: 'transformer.wte.weight'
FAILED tests/test_model.py::test_against_hf_phi_3[device0-dtype0-Phi-4-mini-instruct] - AssertionError: Tensor-likes are not close!

ysjprojects and others added 5 commits March 20, 2025 13:00
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
Co-authored-by: Jirka Borovec <6035284+Borda@users.noreply.github.com>
@ysjprojects
Copy link
Collaborator Author

@ysjprojects mind have look at the failing tests:

FAILED tests/convert/test_hf_checkpoint.py::test_convert_hf_checkpoint[phi-2] - KeyError: 'transformer.wte.weight'
FAILED tests/test_model.py::test_against_hf_phi_3[device0-dtype0-Phi-4-mini-instruct] - AssertionError: Tensor-likes are not close!

Hey @Borda, the first test was fixed.

For the second test, it runs successfully but only when transformers is upgraded to 4.49.0

It's a specific issue with phi-4-mini-instruct that is backward incompatible with previous versions. I have also raised this issue here: #1968

Copy link
Contributor

@lantiga lantiga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, but we're being rate limited by HF on CI tests test_tokenizer_against_hf (on #1952, #1897 as well)

We should cache those assets on our end /cc @Borda

@Borda Borda enabled auto-merge (squash) April 3, 2025 00:52
@Borda Borda merged commit 5c4c09f into Lightning-AI:main Apr 3, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants